[loterre-resolvers] Migrate from GitBucket #46

parmentf · 2024-02-13T12:49:08Z

No description provided.

parmentf · 2024-02-13T12:54:32Z

I forgot that this service get its data from a non-public website.
That makes test to fail, because no initialization succeeds.
Let's decide which way will be used:

dvc
simple webdav
static public server

parmentf · 2024-09-10T14:59:48Z

Decision: use DVC, and update the service each time a resource is updated.
Once migrated, desactivate GitLab CI which heat the cache...

Using their dvc metadata

To be removed when it will work?

And I don't understand why.

To test D63 routes only, debugging purpose

parmentf · 2024-10-09T14:52:22Z

D63's tests don't pass.

Here is the log:

WARNING: uri not found (D63), item #1 was ignored NotFoundError: No cache entry for "D63" found in /tmp/databases/41/D63/db/loterre-vocabulaires-loaded
    at Object.getData [as get] (/app/node_modules/@ezs/storage/node_modules/cacache/lib/get.js:25:11)
    at async Object.load (/app/node_modules/@ezs/storage/lib/load.js:38:19) {
  code: 'ENOENT',
  cache: '/tmp/databases/41/D63/db/loterre-vocabulaires-loaded',
  key: '"D63"'
}

<--- Last few GCs --->

[60:0x4ef6cb0]   227421 ms: Scavenge 1013.4 (1033.4) -> 1010.4 (1033.9) MB, 2.0 / 0.0 ms  (average mu = 0.174, current mu = 0.108) allocation failure 
[60:0x4ef6cb0]   227429 ms: Scavenge 1013.9 (1033.9) -> 1010.9 (1034.1) MB, 2.0 / 0.0 ms  (average mu = 0.174, current mu = 0.108) allocation failure 
[60:0x4ef6cb0]   227438 ms: Scavenge 1014.3 (1034.1) -> 1011.3 (1042.6) MB, 2.2 / 0.0 ms  (average mu = 0.174, current mu = 0.108) allocation failure 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb090e0 node::Abort() [/root/.nvm/versions/node/v16.20.2/bin/node]
 2: 0xa1b70e  [/root/.nvm/versions/node/v16.20.2/bin/node]
 3: 0xce1a20 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/root/.nvm/versions/node/v16.20.2/bin/node]
 4: 0xce1dc7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/root/.nvm/versions/node/v16.20.2/bin/node]
 5: 0xe99435  [/root/.nvm/versions/node/v16.20.2/bin/node]
 6: 0xe99f16  [/root/.nvm/versions/node/v16.20.2/bin/node]
 7: 0xea843e  [/root/.nvm/versions/node/v16.20.2/bin/node]
 8: 0xea8e80 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/root/.nvm/versions/node/v16.20.2/bin/node]
 9: 0xeabd75 v8::internal::Heap::HandleGCRequest() [/root/.nvm/versions/node/v16.20.2/bin/node]
10: 0xe39417 v8::internal::StackGuard::HandleInterrupts() [/root/.nvm/versions/node/v16.20.2/bin/node]
11: 0x11e5ab4 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [/root/.nvm/versions/node/v16.20.2/bin/node]
12: 0x15d9e59  [/root/.nvm/versions/node/v16.20.2/bin/node]

This StackOverflow says that node jave a default value of 512MB of memory.

One should try to increase the setting to more (1GB) using:

export NODE_OPTIONS="--max-old-space-size=1024"

parmentf · 2024-10-09T14:56:05Z

Ooops. config.json already sets the option to 1GB.

web-services/services/loterre-resolvers/config.json

Line 11 in 838453c

"NODE_OPTIONS": "--max_old_space_size=1024",

Let's try 2GB, or more...

parmentf · 2024-10-15T13:06:08Z

Github Action failed:

#28 [build  8/13] COPY --chown=daemon:daemon --from=dvcfiles /dvc/databases /tmp/databases
#28 DONE 694.1s

#29 [build  9/13] COPY --chown=daemon:daemon ./v1 /app/public/v1
#29 DONE 0.1s

#30 [build 10/13] COPY --chown=daemon:daemon ./swagger.json /app/public/
#30 DONE 0.0s

#31 [build 11/13] COPY --chown=daemon:daemon ./config.json /app
#31 DONE 0.0s

#32 [build 12/13] WORKDIR /app/public
#32 DONE 0.0s

#33 [build 13/13] RUN rm /app/public/parse.*
#33 DONE 0.4s

#34 exporting to image
#34 exporting layers
Killed
ERROR: failed to solve: Canceled: context canceled
Error: The operation was canceled.

I fear that the problem is caused by the number of files from stage 8 (databases), because StackOverflow says to add a .dockerignore file to solve the context canceled error. But our .dockerignore is already very restrictive.

find databases/ | wc -l
2067450

2 millions files is big, indeed.

A solution would be to untar the files at container's start, but that means to overwrite the docker-entrypoint.sh from the ezs-python-server base image:

#!/bin/sh

# Restore owner
find /app/public ! -user daemon -exec chown daemon:daemon {} \;
find /tmp ! -user daemon -exec chown daemon:daemon {} \;

cd /app
node generate-dotenv.js
cd /app/public

# Run ezs server as daemon user
su -s /bin/bash daemon -c "npx dotenv -e ../.env -- npx ezs --daemon ./"

Waiting for the uncompressing of databases (.tgz)

They don't exist any more as soon as /tmp volume is mounted.

feat(loterre-resolvers): Migrate from GitBucket

4c2016e

parmentf added the enhancement New feature or request label Feb 13, 2024

parmentf self-assigned this Feb 13, 2024

parmentf mentioned this pull request Mar 27, 2024

fix ezs-python-server: change /tmp owner #68

Merged

23 tasks

parmentf added migration Migrate from GitBucket and removed enhancement New feature or request labels May 31, 2024

parmentf changed the title ~~Migrate loterre-resolvers from GitBucket~~ [loterre-resolvers] Migrate from GitBucket Aug 1, 2024

parmentf added 21 commits September 25, 2024 15:45

chore(vscode): Add Loterre to dictionary

039bd8e

chore(loterre-resolvers): Reduce default concurrency from 4 to 2

0edb863

build(loterre-resolvers): Prepare swagger.json

cb6717f

feat(loterre-resolvers): Add data files

c591eb0

Using their dvc metadata

build(loterre-resolvers): Get data from dvc

e32cf24

style(loterre-resolvers): Spelling

181d265

build(loterre-resolvers): Make /app/data readable

3c0f176

refactor(loterre-resolvers): Use local skos instead of the web one

4e17de0

docs(loterre-resolvers): Add technical explanation

09097c0

To be removed when it will work?

style(loterre-resolvers): Spelling

ba295f1

test(loterre-resolvers): Skip D63, as it does not work

8ff211d

And I don't understand why.

Merge branch 'main' into services/loterre-resolvers/migrate

56720fc

style(loterre-resolvers): Indent test payload

16b38de

chore(loterre-resolvers): Add words to the spelling dictionary

ecb4bb1

fix(loterre-resolvers): singleton alone crashed the script annotate.cfg

3d4a71b

style(loterre-resolvers): Spell

6b5a49b

style(loterre-resolvers): Spelling

717d086

style(loterre-resolvers): Spelling

d780066

refactor(loterre-resolvers): Simplify configuration files

e810ca5

docs(loterre-resolvers): Spelling

1e4d742

refactor(loterre-resolvers): Remove useless fetch.cfg

fa1f910

parmentf added 2 commits October 9, 2024 11:51

chore(loterre-resolvers): Ignore D63.hurl

6835f04

To test D63 routes only, debugging purpose

release [email protected]

838453c

parmentf added 3 commits October 9, 2024 17:05

fix(loterre-resolvers): Fix D63 tests

d9e5298

release [email protected]

d8037c8

test(loterre-resolvers): Add JVR tests

276ef7f

parmentf mentioned this pull request Oct 11, 2024

[Loterre-resolvers] create databases Inist-CNRS/ws-data#8

Merged

parmentf added 4 commits October 15, 2024 11:58

feat(loterre-resolvers): Get databases from DVC remote

7f3c6f7

style(loterre-resolvers): Remove obsolete comments

080749c

feat(loterre-resolvers): Remove weekNumber from databases paths

f9c9231

build(loterre-resolvers): Forgot to set the dockerfile syntax version

53fd7bb

parmentf added 14 commits October 15, 2024 17:14

fix(loterre-resolvers): Uncompress databases in docker-entrypoint.sh

9227206

ci(loterre-resolvers): Increase retry

b34913d

ci(loterre-resolvers): Sleep 1m

9b0c7f3

ci(loterre-resolvers): sleep 2m

59837f0

Waiting for the uncompressing of databases (.tgz)

ci(loterre-resolvers): Increase retry-interval to 30s

e6661c5

ci(loterre-resolvers): Add verbose option to hurl

2c4c640

ci(loterre-resolvers): Add retry-interval to ALL scripts

0600294

refactor(loterre-resolvers): Remove dead code

f1a416f

release [email protected]

a862ac0

fix(loterre-resolvers): Don't store .tgz in /tmp

75ae16e

They don't exist any more as soon as /tmp volume is mounted.

release [email protected]

94a44f6

chore(loterre-resolvers): Update port

4ff72c5

chore(loterre-resolvers): Activate server

bf0065b

Merge branch 'main' into services/loterre-resolvers/migrate

4c80274

parmentf merged commit 1de6e16 into main Oct 23, 2024
1 check passed

parmentf deleted the services/loterre-resolvers/migrate branch October 23, 2024 07:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[loterre-resolvers] Migrate from GitBucket #46

[loterre-resolvers] Migrate from GitBucket #46

parmentf commented Feb 13, 2024

parmentf commented Feb 13, 2024 •

edited

Loading

parmentf commented Sep 10, 2024

parmentf commented Oct 9, 2024 •

edited

Loading

parmentf commented Oct 9, 2024

parmentf commented Oct 15, 2024

[loterre-resolvers] Migrate from GitBucket #46

[loterre-resolvers] Migrate from GitBucket #46

Conversation

parmentf commented Feb 13, 2024

parmentf commented Feb 13, 2024 • edited Loading

parmentf commented Sep 10, 2024

parmentf commented Oct 9, 2024 • edited Loading

parmentf commented Oct 9, 2024

parmentf commented Oct 15, 2024

parmentf commented Feb 13, 2024 •

edited

Loading

parmentf commented Oct 9, 2024 •

edited

Loading